OPR: Partial Deterministic Record and Replay for One-Sided Communication

Deterministic replay of parallel execution and communication operations is required both by HPC debuggers and resilience mechanisms. Despite its potential performance advantages, the inherent nondeterminism present in one-sided communication makes replaying difficult. The essential problem is that the readers of updated shared data do not have any information on which remote threads produced the updates. This paper presents OPR (One-sided communication Partial Record and Replay), the first known software tool for record and deterministic replay for one-sided communication. We have designed OPR from first principles with scalability as its main goal. OPR allows the user to specify a set of tasks of interest and then “records” their execution. The tasks in this set can be replayed, while any other task from the original execution can be abstracted away. OPR provides determinism by using a combination of dataand order-replay. To ensure scalability with the value and the order logs, we carefully optimize the recording stage: values are logged on the first read or only when changed; orderering is imprecisely maintained using a tailored vector clock algorithm. Our evaluation on deterministic and non-deterministic UPC programs shows that OPR introduced an overhead ranging from 1.3× to 27×, when running on 1,024 cores and tracking up